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EarthSat Multispectral Scanner Validation 


Scope of Work 

This report provides results of an independent assessment of the horizontal geopositional 
accuracy of Earth Satellite (EarthSat) Corporation's GeoCover™ orthorecti fied Landsat 
Multispectral Seamier (MSS) imagery. This imagery was purchased through NASA's 
Earth Science Enterprise (ESE) Scientific Data Purchase (SDP) program via NASA 
contract NAS 13-98046. 

Background 

The technique used to validate the geopositional accuracy of EarthSat's MSS imagery 
required the use of validated Landsat Thematic Mapper (TM) imagery. The validated TM 
imagery used in this assessment was also purchased from EarthSat through the SDP. The 
geopositional accuracy of this TM imagery was independently validated by Earth Science 
Applications (ESA) Directorate personnel prior to this effort. 

The TM sensor onboard Landsat 4 and Landsat 5 is a high-spatial-resolution (30-meter 
ground sample distance (GSD)), multispectral system with seven bands in the visible and 
infrared (IR) regions of the electromagnetic spectrum. EarthSat used a two-step process 
to orthorectify the TM imagery used in this assessment. The first step orthorecti fied TM 
scenes for which government-provided ground control points (GCPs) were available. The 
second step used the initially orthorectified scenes as control to tie the remaining raw 
imagery together using a single block adjustment. These orthorectified regions 
correspond to general land areas, or blocks, throughout the world. The result of this 
process is the block-orthorectified set of Landsat TM imagery that was delivered to 
NASA through the SDP. The SDP contract specification for horizontal geopositional 
accuracy of the TM dataset is a net root mean square error ( RMSE net ) of 50 meters. 

SSC ESA Directorate personnel then independently validated the orthorectified TM 
imagery for horizontal geometric accuracy. Additional validation-dedicated, government- 
provided ground control data was used to this end (i.e., the validation-dedicated GCPs 
were separate and independent of the GCPs used in the EarthSat orthorectification 
process). However, because of the limited number of additional government-provided 
GCPs within each scene, the validation assessment was performed by geographic block 
as opposed to individual scenes. Table 1 presents the results of the TM geolocational 
accuracy assessment including the x-component of the root mean square error ( RMSE X ), 
the y-component of the root mean square error ( RMSE y ), and the RMSE nel . Additional 
background and test procedures used in the TM validation effort may be found in the 
publicly available TM validation reports located on the ESA Web site at 
http://www.esa.ssc.nasa.gOv/datapurchase/v v/es/es val reports. asp . 

The MSS sensor onboard Landsat 1 through Landsat 4 is also a high-spatial-resolution 
(57-meter GSD), multispectral system with four bands in the visible and near- IR regions 
of the electromagnetic spectrum. EarthSat orthorectified the MSS imagery used in this 
assessment using an “image mapping" method. This method orthorectified the individual 
MSS scenes to their corresponding orthorectified TM scenes. Unlike the TM 
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orthorectification process, which yielded a block-orthorectified dataset, the result of the 
image mapping process was a set of MSS imagery where each scene was orthorectified 
individually. 


Table 1. TM Horizontal Geometric Accuracy Results 


Geographic Block 


RMSEx (m) RMSEJm) j ftMSE„ et ( m) 


Alaska 


30.57 


33.40 


45.28 


Balkans 


23.17 


21.53 


31.62 


Caribbean 


19.17 


19.87 


27.61 


Central America 


19.44 


15.32 


24.75 


Central Asia 


18.45 


27.53 


33.14 


Central North America 


19.33 


18.56 


26.80 


East Africa 


20.88 


18.97 


28.21 


Eastern North America 


18.66 


18.90 


26.56 


Europe 


24.46 


Middle East 


32.84 


26.64 


29.12 


36.16 


43.89 


North Africa 


28.95 


38.28 


48.00 


Northeast Asia 


24.42 


25.61 


35.39 


Northwest Asia 


35.81 


27.39 


45.09 


South Africa 


17.86 


19.22 


26.24 


Southeast Asia 


24.38 


26.45 


35.97 


Southern South America 


1.58 


2.40 


2.87 


Western North America 


16.08 


14.45 


21.62 


SSC ESA Directorate personnel validated the geopositional accuracy of the orthorectified 
MSS imagery by performing a relative assessment between the TM and MSS imagery. In 
other words, the geopositional accuracy of the MSS scenes was determined relative to 
their corresponding validated TM scenes. The TM images used in this investigation were 
the same images used in the original TM validation assessment. The rationale for this 
approach is explained in the following section. 


Approach 

To understand the MSS validation approach, it is useful to define the TM and MSS 
horizontal accuracy specifications. The geometric accuracy specified for Landsat TM 
imagery procured through the SDP is an RMSE net of ±50 meters as specified by NASA 
contract NAS 13-98046 with Earth Satellite Corporation. Similarly, the geometric 
accuracy specification for Landsat MSS imagery procured through the SDP is an RMSE„ et 
of ±100 meters. 

The coarser spatial resolution of the MSS orthorectified products did not permit the 
identification of the same ground control features used for the TM accuracy assessment. 
Thus, a direct absolute accuracy assessment using identifiable GCPs was not possible. 
Additionally, because EarthSat orthorectified the MSS imagery on a scene-by-scene basis 
using TM imagery as horizontal control, a scene-by-scene assessment approach for the 
MSS imagery was identified as more appropriate than the regional block assessment used 
in the TM validation. In addressing these two issues, it became clear that a relative 
geometric assessment between the previously validated TM imagery and the MSS 
imagery was the best available approach for this assessment. 


Page 3 of 27 



EarthSat Multispectral Scanner Validation 


To verify the accuracy of the MSS imagery, selected MSS scenes were compared to the 
corresponding verified/validated TM scene. In this analysis, it was assumed that if a 
particular TM scene was found to meet the =50-meter RMSE nel specification, that scene 
could serve as "truth" for verification of the corresponding MSS scene for the same area. 
The results of the SSC ESA Directorate independent TM validation are shown in Table 1 . 
This table shows that all of the TM geographic blocks meet the ±50-meter horizontal 
accuracy specification and are viable control sources for the MSS geometric 
verification/validation effort. The locations of identifiable features in MSS scenes were 
compared to the locations of the same features within the TM scenes. Because the 
method for assessing the geometric accuracy of MSS imagery is relative to the previously 
validated TM imagery, it was determined that as long as the RMSE ne t of the MSS imagery 
was <50 meters, then the imagery would be within specification. If the imagery produced 
an RMSEnet >50 meters, a worst-case scenario investigation into the TM and MSS 
imagery would ultimately determine the success or failure of the scene to meet the 
specification. 

Dataset 


The limited number of TM scenes having validation-dedicated, government-provided 
control data ultimately determined the size of the dataset that could be used in the 
Landsat MSS geopositional accuracy assessment. A total of 91 Landsat MSS scenes, each 
corresponding to validated Landsat TM scenes, were used in this assessment. 
Appendix A contains a map showing the location of the scenes used in relation to the 
entire globe. 

Procedure 


The relative horizontal geopositional accuracy assessment of MSS imagery began with 
the identification of MSS imagery that corresponded to validated TM imagery. Next, the 
data were analyzed using the GCP editor function available in ERDAS IMAGINE image 
processing software. For each scene, the Landsat MSS and corresponding TM images 
were opened in separate viewers. At this point both the MSS and TM images were 
inspected for clearly identifiable features. When specific landmarks, such as rock 
formations, roads, and waterways, were identifiable in both scenes, these landmarks were 
selected and geographically located. This action populated the X-Input and Y-Input fields 
in the software's GCP editor. The Universal Transverse Mercator (UTM) coordinates 
associated with each landmark in the MSS scene were compared to the UTM coordinates 
associated with each corresponding landmark in the TM scene. 

After locating a minimum of 20 usable points within the MSS-TM scene pair, the analyst 
exported an American Standard Code for Information Interchange (ASCII) file that 
contained the Point IDs, input coordinates, and reference coordinates for each scene 
examined. These individual files provided the required input into the SSC ESA 
Directorate Visual Basic program written to compute relative geopositional accuracy. For 
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each point, the Visual Basic program computed the X-differences and Y-differences, the 
squares of the X-differences and Y-differences, the RMSE X and RMSE y , and the RMSE„ el . 


RMSE 

RMSE 

RMSE 


input ^ control / 


yfr -y )- 

l—U \ input control / 


’ \ n 

_ =x(RMSEj +(RMSE~) : 


where: 

Xi n p Uh y input are the coordinates of the MSS input points, 
Xcnmroin V control are the coordinates of the TM reference points, 
n is the total number of usable control points 

Limitations 


Because the validated Landsat TM imagery is treated as a “truth” dataset for the MSS 
validation, it introduces limitations in the MSS validation approach. First, the same 
dataset is used in the EarthSat orthorectification process and the independent accuracy 
validation process, so the validation method is not truly independent. Second, any 
accuracy errors in the TM imagery can potentially have adverse affects on the absolute 
geometric accuracy of the validated MSS imagery. Appendix B provides a cartographic 
representation showing the locations of the TM scenes for each Landsat block used in 
validating the orthorectified MSS imagery. Additionally, each cartographic representation 
provides the resultant error vectors of all of the scenes from the NASA ESA Directorate’s 
assessment of TM imagery. The resulting vector length represents the relative magnitude 
of error. The direction of the resulting vector represents the overall direction of the error. 
This cartographic output was derived from a government-provided vector-based program, 
“GeoCover Analysis.” 

The MSS validation approach was also dependent on the geographic distribution of the 
available MSS-TM scene pairs used in the assessment. This distribution is shown in 
Appendix A. This validation procedure assumes that the scenes selected represent a large 
enough sample of imagery to be extrapolated to the rest of the MSS imagery covering the 
globe. 

The possibility exists for inherent analyst bias because of the variability in image 
interpretation and pattern recognition capabilities. Final selection of points reflects user 
subjectivity. 
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Results 


In 87 percent of the scenes, the RMSEnet was <50 meters or the scenes yielded results 
such that the observed geolocational accuracy passed a worst-case scenario test 

(•j c RMSE, ir „ + MSS RMSE npr — 1 00 meteis ^ p our percent of the imagery was unable 

to be evaluated for different reasons, including extreme scene uniformity. The remaining 
9 percent of the imagery failed to meet the specification in a worst-case scenario 

(-j e TM RMSE nct + MSS RMSE ne , > 1 00 meters ^ g p er cent, 4 images violated the 

RMSEnet worst-case scenario by <10 meters. Table 2 depicts a brief summary of the 
results. Table 3 depicts details on the scenes that failed the worst-case scenario. 


Table 2. MSS Validation Pass/Fail Metrics 


Pass/Fail Criteria 

Number of 
Scenes 

Percentage of 
! Scenes 

Passed 

80 

87% 

Failed: Exceeded worst-case absolute RMSE net 

8 

9% 

Scenes unable to validate 

4 

4% 1 

Total Scenes 

92 



Block 

TM 

Path 

TM 

Row 

MSS 

Path 

MSS 

Row 

TM 

RMSE ne , 

(m) 

MSS 

RMSE net 

(m) 

Worst-Case 

RMSE n et 

(m) 

East Africa 

176 

55 

189 

55 

28.21 

110.45 

138.66 

East Africa 

178 

46 

191 

46 

28.21 

72.82 

101.03 

Eastern 
North America 

22 

39 

23 

39 

26.56 

75.62 

102.18 

Europe 

198 

23 

213 

23 

36.16 

71.08 

107.24 

North Africa 

196 

34 

211 

34 

48.00 

60.89 

108.89 

Southeast Asia 

130 

50 

140 

50 

35.97 

86.43 

122.40 

Southeast Asia 

130 

52 

139 

52 

35.97 

85.15 

121.12 

Southeast Asia 

133 

48 

143 

48 

35.97 

90.55 

126.52 


The data results for each individual scene are recorded in Appendix C. These results 
record the Path/Row of the Landsat MSS and TM imagery as well as the RMSEx, 
RMSEy, and the RMSEnet for each scene. 


Conclusions 


Given the limitations inherent in this analysis, it is evident that to a great extent the MSS 
imagery meets specifications. EarthSat has been notified of the few scenes that failed to 
meet the geolocational specifications or were unable to be validated. For various reasons, 
EarthSat was willing to accept the failures of the 8 scenes and the inability to validate the 
remaining 4 scenes for the purposes of this report. No further deliveries of validation data 
are expected at this time. 
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Northwest Asia Landsat TM Scene Errors 
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South Africa Landsat TM Scene Errors 
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Southeast Asia Landsat TM Scene Errors 
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Southern South America Landsat TM Scene Errors 
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Appendix C 

Landsat MSS Scene Accuracy Computations 


! 

Block 

TM Path 

1 

TM Row 

1 

MSS Path 

MSS Row 

\ 

RMSE X 

(m) 

RMSE y 

(m) 

RMSE net 

(m) 

Alaska 

i 6 

11 

7 

li If 

27.76 

26.86 

38.57 

Alaska 

I 6f 

E 

7 : 

U 

29.85 

23.76 

38.13 

Alaska 

6( 

If 

7 : 

16 

18.6' 

18.78 

26.43 

Alaska 

§6 

41 

i Of 

44 

r ' ' . 

- " 


Alaska 

6~ 

If 

7 : 

18 

14.66 

25.56 

29.49 

Alaska 

6$ 

U 

71 

Si 14 

25.61 

28.17 

38.07 

Alaska 

6$ 

If 

7t 

5 16 

28.44 

25.66 

38.32 

Alaska 

j 7C 

16 

76 

16 

25.66 

26.37 

36.79 

Alaska 

72 

17 

78 

17 

30.86 

22.68 

38.30 

Alaska 

72 

18 

78 

18 

23.32 

27.37 

35.96 

Balkans 

166 

22 

178 

22 

32.41 

22.80 

39.63 

Balkans 

171 

24 

184 

24 

30.77 

29.11 

42.36 

Balkans 

179 

20 

193/194 

20 

28.26 

28.96 

40.46 

Balkans 

181 

19 

195 

19 

34.58 

29.88 

45.70 

Balkans 

183 

19 

198 

19 

27.78 

37.07 

46.32 

Balkaog 

UQS r\ CM ID 

4 33 

20 

407 

20 

- - 


... ' , - 

Caribbean 

11 

45 

12 

45 

33.09 

30.82 

45.22 

Caribbean 

12 

46 

12 

46 

32.75 

31.98 

45.77 

Caribbean 

16 

44 

17 

44 

31.40 

23.48 

39.20 

Central America 

35 

38 

37/38 

38 

22.12 

30.28 

37.50 

Central Asia 

128 

24 

138 

24 

41.22 

37.59 

55.79 

Central Asia 

130 

24 

140 

24 

34.19 

38.12 

51.20 

Central North America 

24 

39 

26 

39 

35.24 

31.16 

47.04 

Central North America 

27 

33 

29 

33 

32.50 

33.80 

46.89 

Central North America 

32 

36 

34 

36 

30.99 

29.65 

42.89 

Central North America 

34 

32 

37 

32 

28.15 

33.45 

43.72 

East Africa 

161 

53 

173 

53 

28.81 

31.30 

42.54 

East Africa 

166 

63 

178 

63 

27.49 

17.39 

32.53 

East Africa 

167 

55 

179 

55 

29.64 

25.13 

38.86 

East Africa 

167 

64 

179 

64 

23.40 

23.43 

33.11 

East Africa 

169 

50 

182 

50 

28.94 

27.23 

39.74 

East Africa 

169 

54 

181 

54 

27.76 

36.33 

45.72 

East-Afrtea 

470 

§§ 

482 

00 

■ ' -• 

>7 "- 77 - 


East Africa 

173 

51 

186 

51 

29.74 

38.20 

48.41 

East Africa 

173 

57 

186 

57 

34.42 

43.14T 

55.19 

East Africa 

174 

38 

187 

38 

28.53 

28.32 

40.20 

East Africa 

175 

49 

188 

49 

31.46 

39.23 

50.29 

East Africa 

176 

55 

189 

55 

82.28 

73.69 

110.45 

East Africa 

178 

46 

191 

46 

51.43 

51.55 

72.82 

East Africa 

179 

49 

192 

49 

30.47 

25.24 

30.47 
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Block 

TM Path 

TM Row 

MSS Path 

MSS Row 

| RMSE X 
(m) 

RMSEy 

(m) 

RMSE n e t 

(m) 

East Africa 

! 165(164 

)i 5C 

177 

53 

21.39 

26.16 

33.80 

Eastern North America If 

22 

16 

29 

40.81 

i 27.03! 48.95; 

Eastern North America 1 If 

41 

17 

'! 41 

48.54 

45.29! 66.39| 

iEastern North America 18 37 19 37 

28.31 

31.49! 42.64! 

Eastern North America 

21 

3^ 

• 23 

34 

22. 7C 

23.66 

>! 32.77 

Eastern North America 

22 

39 

23 

39 

64.79 

38.99 

75.62' 

Europe 

i 185 

17 

199 

! 17 

| 44.65 

33.1C 

55.58j 

d irr\r\a 

1 QC 

35 

19g 

32 




Europe 

188 

23 

203 

23 

28.46 

35.49 

45.49 

Europe 

190 

29 

205 

29 

35.36 

27.47 

44.77 

Europe 

192 

26 

207 

26 

28.89 

26.04 

38.89 

Europe 

194 

24 

209 

24 

25.47 

24.78 

35.53 

Europe 

198 

23 

213 

23 

48.68 

51.79 

71.08 

Europe 

200 

28 

216 

28 

25.49 

16.20 

30.20 

Europe 

201 

26 

217 

26 

35.43 

36.51 

50.87 

Middle East 

159 

40 

171 

40 

29.14 

37.72 

47.66 

Middle East 

159 

41 

171 

41 

25.77 

32.72 

41.64 

Middle East 

163 

40 

175 

40 

29.72 

40.81 

50.49 

Middle East 

166 

37 

178 

37 

24.23 

32.93 

40.89 

Middle East 

168 

38 

181 

38 

26.86 

26.06 

37.42 

Middle East 

171 

36 

184 

36 

29.29 

33.14 

44.23 

North Africa 

196 

34 

211 

34 

31.52 

52.10 

60.89 

Northeast Asia 

114 

34 

123 

34 

25.17 

32.85 

41.38 

Northeast Asia 

115 

31 

124 

31 

36.64 

30.75 

47.84 

Northeast Asia 

116 

34 

125 

34 

27.841 

25.46 

37.72 

Northeast Asia 

116 

36 

125 

36 

25.20 

24.80 

35.36 

Northeast Asia 

117 

33 

126 

33 

24.87 

26.91 

36.64 

Northeast Asia 

126 

25 

136 

25i 

33.90 

28.56 

44.33 

Northwest Asia 

146 

22 

157 

22 

32.62 

30.57 

44.71 

South Africa i 177: 

75 

1 90 

75! 

33.84 

43.88 

55.41 

i r 

South Africa 

177 

76 

190 

76 

29.75 

27.09 

40.27 

South Africa 

178 

75 

191 

75 

35.36 

46.09 

58.09 

South Africa 

178 

76 

191 

76 

47.03 

33.86 

47.03 

South Africa 

179 

75 

192 

75 

26.37 

30.17 

40.07 

South Africa 

179 

76 

192 

76 

23.92 

28.66 

37.33 

Southeast Asia 

117 

45 

126 

44 

40.90 

35.75 

54.33 

Southeast Asia 

119 

42 

127 

42 

26.99 

32.48 

42.23 

Southeast Asia 

125 

52 

134 

52 

26.88 

24.79 

36.56 

Southeast Asia 

127 

48 

1 36^ 

48 

32.95 

33.20 

46.78 

Southeast Asia 

128 

56 

137 

56 

22.49 

25. 78 

34.21 

Southeast Asia 

130 

50 

140 

50 

81.98 

27.38 

86.43 

Southeast Asia 

130 

52 

139 

52 

82.00 

22.97 

85.15 

Southeast Asia 

131 

46 

140 

46 

27.29 

24.07 

36.39 

Southeast Asia 

133 

48 

143 

48 

62.76 

65.27 

90.55 
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Block 

TM Path 

TM Row 

MSS Path 

MSS Row 

RMSEx 

(m) 

RMSEy 

(m) 

RMSE net 

(m) 

Southeast Asia 

134 

45 

144 

45 

30.34 

32.99 

44.82 

Southern South America 

226 

78 

242 

78 

52.13 

34.92 

62.74 

(Western North America 

38 

37 

41 

37 

38.49 

24.70 

45.73 

j i 

IWestern North America 

40! 

37 

43 

37 

28.57, 

31.63 

r 42.62 

IWestern North America , 

41 

34| 

44 

34 

23.74 

30.61 

38.74 

Iwestern North America j 

42! 

25| 

45! 

25 

51.52 

52.26 

73.38 

Western North America j 

46! 

27! 

50' 

27; 

27.17| 

36.72! 

45.68! 


Note: Rows highlighted in grey indicate that the imagery analysis was unable to be completed. Rows with red text 
indicate that the data failed to meet specification. 
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